Modeling and Results Supplement
Purpose of the Document
The aim of these supplementary details are to provide readers with better documentation of the complete modeling and model selection process. There were a total of 11 models fit as part of this study, and the number of analyses and supporting figures to understand this process are simply too long for a single document. This document therefore provides relevant plots and summaries to help readers understand the steps taken to arrive at the final model, and perhaps more importantly, the document provides commentary for how each modeling choice was made.
The goal of this document is therefore to ensure complete transparency in the data analysis process. Accompanying this document is also the R script file that was created during the study process. Someone who acquires access to the appropriate files through the HRS should be able to run the script file (with some minimal alterations to ensure that the files are read from the correct files) and then get results that match those shown here (within some margin of error due to randomness in some of the methods, though the use of the same seed in R and brms should reduce the impact of this).
Overview of Sections
In an attempt to facilitate readability of this document, the modeling process is broken down into a few different sections. Additionally, there is a table of contents that readers can use to jump to any section, so for those interested in specific aspects of this supplementary material, it is useful to know what exists in each section.
First, a flowchart showing the order in which models were built and the model to which each was tested is provided. Each candidate model was compared to the up-to-then best fitting model. The aim of this flowchart is to help familiarize readers with the models and modeling steps quickly and efficiently as the remaining sections add greater detail.
Second, the priors used in the models are explored. As the data analyses were done in the Bayesian framework, the analysis of the priors is useful. The priors are shown as well as the prior predictive checks for a fixed item model.
Third, the details of each model are provided. This is the largest section as each model includes a variety of explorations. To help reduce the overall length of the document, each model is given its own tab so that readers can select one model at a time. Details for each model can be broken down into 3 primary sections: model validity, model performance, and model estimates.
- Model validity refers to tests of whether the estimation process converged and not subject to any issues that would make estimates from the model entirely invalid or unstable. These tests include visual inspection of chain mixtures, \(\hat{R}\), effective sample size, and maximum treedepth.
- Model performance refers to the posterior predictive checks that mirror those shown in the manuscript for the final model: predictions of all responses for all items, responses to each item, and responses to all items for a subset of participants. As the space of this document is less limited than space in the final manuscript, the random subset of participants is increased from 12 to 20.
- Model estimates refers to the summary of model parameters like fixed and random effects estimates. This summary is different than the one presented in the manuscript for the final paper as the objective of these intermediary models is not to summarize effect sizes or the probability of these effects; instead, the goal is to get a general idea of what the model is estimating and how it is performing. Toward this end, conditional effects plots for each model are also included. Note that these plots may not be very informative for the majority of the models tested because only a subset included many covariates.
Finally, as discussed in the manuscript, some additional details regarding the final model are also included here. These details include summaries of the item parameters in traditional IRT metrics (i.e., as difficulty and discrimination), the reliability plot, expected score functions (total test), and information functions (total test and all trials). As in the manuscript, these item-specific plots are provided with and without assuming previous learning of the words. The test is best understood as a dynamic one in which a person’s performance on each trial changes our expectation for how they will perform on the next.
What’s Being Done
As mentioned earlier, this document shows all the code used to generate the results. Since there is an accompanying R data script, it may be useful for readers to know the objects being called in this Markdown document since those objects can be connected back to the R script. The hope is that this will create a reasonable sense of cohesion between the supplementary materials, and it should mean that all the results here are also fully reproducible. Toward that end, the objects and packages used in this document are shown below (note that the R objects are read in as RDS files whose names are consistent with those listed in the R script file).
#read in needed data
df_long <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Saved Objects for RMDs/Item Covariate Analyses/Data_long.rds")
Rasch_prior <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/1PL_prior_check.rds")
TwoPL_prior <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_prior_check.rds")
Rasch_inter <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/1PL_intercept.rds")
TwoPL_inter <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_intercept.rds")
TwoPL_learn <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_growthModel.rds")
TwoPL_multi <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_multidimensional.rds")
TwoPL_chnge <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_changeModel.rds")
TwoPL_depmd <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_dependencyModel.rds")
TwoPL_depun <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_dependencyUniqueModel.rds")
TwoPL_deptr <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_dependencyTrialModel.rds")
TwoPL_itmex <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_itemCovariates.rds")
#load in required packages
#wrapping in suppressPackageStartupMessages() done just to reduce print out in document
suppressPackageStartupMessages(library(brms))
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(kableExtra))Modeling Flowchart
As can be inferred from the figure below, the modeling process involved several iterations of fitting similar models and considering results from old models. As there specific hypotheses regarding the model and its covariates, the models needed to test these hypotheses were naturally created as part of the study. At the same time, alternative models had to be specified against which to test these models. Additionally, while there were hypotheses regarding what covariates (and their signs) would ultimately be in the final model, the important result of the study is the final model itself. Some readers may see the number of models examined and variations in their specification and become concerned of potentially two things: multiple comparisons + inflated error rates and/or model fishing/p-hacking.
The objective of this supporting document is to help clarify all the modeling choices so that readers do not need to question whether model specifications were made to try to improve performance of the final result. With respect to possible concerns regarding multiple comparisons, Bayesian methods do no suffer from these concerns (Gelman et al., 2013; Gelman, Hill,& Yajima, 2012; Neath, Flores, & Cavanaugh, 2017; Sjölander & Vansteelandt, 2019). While there are several reasons that this is the case for Bayesian methods, it is sufficient to speak to three. First, we do not use null hypothesis testing in this study. Model comparisons are completed using a formal comparison of information criteria to select models with better out-of-sample performance. Coefficients are not interpreted as significant or not but instead are summarized in terms of their probability of existing. Since the correction to p-values for multiple comparisons is to control the risk of falsely rejecting the null hypothesis, this is not a concern when we are not rejecting null hypotheses. Second, we utilize skeptical priors for our effects estimates. This means that we are a priori placing greater probability of the effects being 0 (or practically equivalent to 0). This is the inverse of frequentist decision-making practices where the null hypothesis is very easy to reject since it is constrained to (usually) a nill point value, which is rarely a realistic threshold for any model specification. Finally, the models include skeptical priors, meaning that all effects estimates are pulled closer to those a priori small effects.
Flowchart of all models fit and compared in order (click to make larger)
Prior Specifications and Inspection
As discussed in the manuscript for this study, prior specification came from documentation on using brms for IRT (i.e., Bürkner, 2020a and Bürkner, 2020b). As a general note, the non-linear specification for the 2PL model used in this study come from the Bürkner (2020b) study published in the Journal of Intelligence. Also as discussed in the manuscript, the specification of priors follows the recommendations of other typical multilevel regression guides (e.g., Gelman & Hill, 2007). Specifically, the priors are normal distributions with wide variances relative to the scale of the outcome data. As these priors on the regression coefficients, normal distributions are appropriate prior distributions. While these distributions are centered on 0, they are made wide and thus only weakly informative to the final parameter estimates. This specification helps regularize estimation (i.e., pull estimates toward zero and away from more extreme values) while imparting little a priori influence on the estimates. Additionally, by making the priors skeptical (i.e., they place the greatest probability on very small to non-existent effects), there is a reduction in the risk of experimenter bias; however, with 1219 participants each observed 30 times, the data will dominate the prior anyway.
The priors for the Rasch and 2PL models are shown below:
Rasch_priors <-
prior("normal(0, 5)", class = "b") +
prior("normal(0, 3)", class = "sd", group = "ID")
TwoPL_priors <-
prior("normal(0, 5)", class = "b", nlpar = "beta") +
prior("normal(0, 1)", class = "b", nlpar = "logalpha") +
prior("normal(0, 1)", class = "sd", group = "ID", nlpar = "theta")Readers following the R script file will recognize that the above are repeated in that document (“3 - Measurement and Explanatory Model Fit”, lines 19-27). To read these priors, it can be helpful to look at a couple of examples. Starting with the priors for the Rasch model, the prior for the coefficients (class = "b") is specified as a normal distribution with a mean of zero and standard deviation of five. This would mean that, before looking at the data, we are guessing that there is a 68% probability of the intercept being between -5 and 5 (i.e., +/- 1SD), and we are guessing that there is about a 95% probability that the coefficients will be between -10 and 10 (i.e., +/- 2SD). In these models, the coefficient corresponds to the actual item parameters (for the Rasch, this is item easiness), so these are the ranges in which we are saying are potential values we might observe. Take another example but this time from the 2PL model and a random effect. The random effect priors are all labeled as class = "sd" since we are putting a prior belief on the plausible values of the standard deviation of the random effects. For the random person effect (i.e., latent trait of each participant), we look for the variable that defines this group (group = "ID", where ID is a number indexing each participant) and the non-linear element it is estimating (nlpar = "theta" where \(\theta\) is the traditional IRT simple for the latent trait). The prior therefore for the latent trait is the standard normal distribution with mean of zero and standard deviation of 1. This specification is consistent with the treatment of the latent trait in IRT as normally distributed and z-scaled, though generally in IRT models the variance is constrained to be 1 for identifiability purposes (see Bürnker, 2020a for details).
While it’s comforting to be able to go through each prior specification and think about what it means, it is perhaps more efficient to examine some plots. The first set of plots shown are the parameter estimates returned when the model samples only from the prior. In other words, these are the estimated effects implied by the priors. If the prior specifications are truly skeptical and weakly informative, then they will give the greatest probability to effects of very small size and cover a wide range of plausible values. The prior distribution for the Rasch easiness parameter of “Butter” on trial one.
plot(Rasch_prior, combo = c("dens", "intervals"), variable = "b_ItemButter1", ask = FALSE)
Consistent with expectations, these plots show wide ranges of plausible values with the greatest probabilities being placed on small effects. In the left column are the density plots of the estimates while the right column shows the interval estimates with the circle = mean estimate, bold line = 50% credible interval, and thin line = 90% credible interval. The prior density very clearly follows a normal distribution that allows admissible values with more extreme estimates being given small overall probability of being true. These wide priors allow the data to dominate the posterior estimates for the parameters, though again this would likely be the case with even more informative priors due to the size of the sample available.
plot(TwoPL_prior, combo = c("dens", "intervals"), variable = "b_beta_ItemButter1", ask = FALSE)
The plot above mirrors what was shown for the Rasch model but are for the 2PL model, which is clearly not different as the same priors are used for this parameter. Though all the prior distributions for the model can be shown, this results in many different plots that can be overwhelming to interpret, and these plots correspond to just priors for those specific parameters. Since researchers may not often have explicit expectations for what each parameter will be or should be, these plots are not always directly helpful.
Another related graphical output is the prior predictive check. The prior predictive check runs the model using the priors rather than the observed data (note that this model would be similar to a null model). If the priors are specified well, then they should return reasonable, albeit wide, estimates of the observed data. This plot is more intuitive for understanding the effect of priors instead of parameter-by-parameter plots like above. The prior predictive checks for the Rasch and 2PL models are shown below following the same layout as the posterior predictive checks in the manuscript and for the other models.
pp_check(Rasch_prior, ndraws = 25, type = "bars")
pp_check(Rasch_prior, ndraws = 25, type = "bars_grouped", group = "Item")
pp_check(Rasch_prior, ndraws = 25, type = "bars_grouped", group = "ID",
newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
The Rasch prior predictive checks above demonstrate that the prior specifications are adequately wide to provide coverage for the observed data. The estimates themselves are expectedly poor, which is a product of the skeptical priors. The same plots are now repeated but for the 2PL model.
pp_check(TwoPL_prior, ndraws = 25, type = "bars")
pp_check(TwoPL_prior, ndraws = 25, type = "bars_grouped", group = "Item")
pp_check(TwoPL_prior, ndraws = 25, type = "bars_grouped", group = "ID",
newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
Performance of the 2PL priors are similar to those of the Rasch, suggesting that these priors are also appropriately specified.
While all the above plots and theoretical justifications suggest that the priors are specified consistent with the wishes for the model, it can also be helpful to perform a post-hoc test of whether a model’s priors were influential on its final estimates. As has been mentioned multiple times in this section, due to the sample size of this study, it is expected that the data and not the prior will dominate the posterior estimates, meaning that even with more informative priors the data would have more influence on the final estimates. One such comparison discussed by Andrew Gelman (link here) is to compare the posterior standard deviation (i.e., precision of the effect estimate after looking at the data) to the prior standard deviation (i.e., uncertainty of the effect estimate before looking at the data). In the case that a prior is influential, the ratio of the precision to uncertainty will be large. Put another way, we learn little more about the posterior from observing the data because the prior was already highly informative. Gelman’s recommended threshold for determining whether a prior is informative is if the posterior standard deviation for an effect is more than 0.1 times the prior standard deviation. The able below provides this metric for each predictor from the final model reported in the study.
#get the posterior samples from the final model
posteriors <- as_draws_df(TwoPL_itmex)
#get the fixed effects for the item easiness
beta <- posteriors %>%
select(starts_with("b_beta")) %>%
apply(., 2, function(x) sd(x)/sd(posteriors$prior_b_beta)) %>%
as.matrix()## Warning: Dropping 'draws_df' class as required metadata was removed.#do the same for item discrimination
alpha <- posteriors %>%
select(starts_with("b_logalpha")) %>%
apply(., 2, function(x) sd(x)/sd(posteriors$prior_b_logalpha)) %>%
as.matrix()## Warning: Dropping 'draws_df' class as required metadata was removed.#combine into a single result
result <- rbind(beta, alpha) %>%
as.data.frame() %>%
add_column("Prior Influence" = ifelse(.[, 1] >= 0.1, "Informative", "Uninformative")) %>%
rename("Ratio (Posterior:Prior)" = V1)
row.names(result) <- paste(rep(c("Easiness:", "Discrimination:"), each = 11), rep(c("Intercept", "Trial 2", "Trial 3", "Item Pos. (linear)", "Item Pos. (quadratic)", "Word Frequency", "Concreteness", "Semantic Diversity", "Age of Acquisition", "Body-Object Integration", "Phonemes"), 2))
rm(posteriors, beta, alpha)
#get resulting table
result %>%
kable(caption = "Comparison of the Posterior to Prior Distribution Standard Deviations", digits = 4, align = 'cc') %>%
column_spec(1:3, bold = ifelse(result$`Ratio (Posterior:Prior)` >= 0.10, TRUE, FALSE)) %>%
kable_classic(full_width = FALSE, position = "float_right")| Ratio (Posterior:Prior) | Prior Influence | |
|---|---|---|
| Easiness: Intercept | 0.0122 | Uninformative |
| Easiness: Trial 2 | 0.0146 | Uninformative |
| Easiness: Trial 3 | 0.0153 | Uninformative |
| Easiness: Item Pos. (linear) | 0.7991 | Informative |
| Easiness: Item Pos. (quadratic) | 0.7709 | Informative |
| Easiness: Word Frequency | 0.0088 | Uninformative |
| Easiness: Concreteness | 0.0125 | Uninformative |
| Easiness: Semantic Diversity | 0.0125 | Uninformative |
| Easiness: Age of Acquisition | 0.0105 | Uninformative |
| Easiness: Body-Object Integration | 0.0127 | Uninformative |
| Easiness: Phonemes | 0.0093 | Uninformative |
| Discrimination: Intercept | 0.6915 | Informative |
| Discrimination: Trial 2 | 0.0628 | Uninformative |
| Discrimination: Trial 3 | 0.0662 | Uninformative |
| Discrimination: Item Pos. (linear) | 0.9746 | Informative |
| Discrimination: Item Pos. (quadratic) | 0.9916 | Informative |
| Discrimination: Word Frequency | 0.0407 | Uninformative |
| Discrimination: Concreteness | 0.0575 | Uninformative |
| Discrimination: Semantic Diversity | 0.0586 | Uninformative |
| Discrimination: Age of Acquisition | 0.0532 | Uninformative |
| Discrimination: Body-Object Integration | 0.0515 | Uninformative |
| Discrimination: Phonemes | 0.0459 | Uninformative |
The table of these posterior and prior comparison results is shown here. For convenience, only those whose threshold exceeds the recommended > 0.10 ratio are bolded. Generally, the findings suggest that the priors performed as expected: they were weakly informative and did not seemingly have undue influence on the posterior estimates. Notably, the exceptions are the discrimination intercept, which had a 95% credible interval that included zero, and the item position coefficients that had relatively larger standard errors. This is not unexpected as the result essentially indicates that even after looking at the data we do not change our prior beliefs, which were that the effects were either zero or very wide. The influence of the prior on these null findings reflects a point made earlier regarding how Bayesian methods are generally unaffected by multiple comparisons.
An important point to emphasize at this juncture is the implication of a “significant” finding in Bayesian methods. As discussed throughout this section on priors, the priors here are skeptical of an effect in the sense that they place greatest weight on an effect estimate of zero or close to zero and ambivalence regarding the direction of the effect (i.e., it is just as equally probable that the sign is positive of negative). In the context of the current study, this means that, despite the hypotheses regarding the presence and direction of specific effects, the priors for these predictors are specified in this skeptical way so as to avoid the introduction of experimenter bias. In regard to the robustness of the effects observed, the fact that they are observed from the information provided by the data despite these skeptical priors also helps build confidence in the presence of these effects.
Model Details
The details of this section highlight the models fitting and results. These details speak to the validity of the model results and then also the actual results (i.e., parameter estimates) of the model. Model validity is particularly important in Bayesian methods because the parameter estimates are based on Monte Carlo Markov Chains (or Hamiltonian Monte Carlo (HMC) in the case of these models run using Stan). In cases where a model fails to converge or throw errors under the estimator, the validity of the model results are questionable or even completely invalid (e.g., in the case of divergent transitions). To reflect this need to first confirm the validity of the results, various diagnostics of the model fit are provided first before then presenting the model results.
For readers unfamiliar with these model checks, a brief overview of each is provided here. The largest threat to model results in the HMC is arguably the presence of divergent transitions. HMC explores the posterior distribution by simulating the evolution of a Hamiltonian system, and in order to do this efficiently, the sampler finds a reasonable step size with which to explore that space. A divergent transition occurs when the trajectory of the system is lost due to too large of a step size. Another important model check is the treedepth of the chains. Again, to improve the efficiency of the posterior sampling, a maximum treedepth is set to prevent the estimator spending excessive time in certain steps and spaces. Since this treedepth may artificially limit the estimator in exploring the posterior space, it is important to check whether any of these treedepths were actually hit during estimation (default treedepth is 10). Another important Bayesian model indicator is \(\hat{R}\) because multiple HMC (and MCMC) chains are needed to ensure that the posterior is sampled appropriately. If a single chain is run, then it is not possible to determine whether the random starting values of this chain may have lead to a specific set of parameter estimates. Running multiple independent chains that each have different random starting values helps ensure that the parameter estimates are not biased by exploration of only certain posterior values. In well-behaved models, these chains will mix together without any clear indications of one chain producing a specific set of parameter estimates that differ from what other chains are estimating. While this mixture of chains can be visually inspected via the trace plot (also provided here), the \(\hat{R}\) statistic is a simple indicator of this with the conservative recommendation of treating estimates as valid only if the \(\hat{R}\) for the parameter is less than 1.01. A final model validity check shown here is the effective sample size. Because multiple chains are run for many samples of the posterior, it is expected that some of those samples are autocorrelated and thus dependent on previous samples. Effective sample size informs us of the precision of the model in MCMC and HMC methods. When samples are independent, the central limit theorem indicates that the precision with which a parameter can be estimated is proportional to the size of the sample (e.g., \(\sigma_\bar{x} = \frac{\sigma}{\sqrt{N}}\)). The same proportionality can be obtained when samples are dependent but requires replacing \(N\) with \(N_{ESS}\), or the effective sample size. Due to the dependence of the sampling, \(N_{ESS} < N\) and thus precision of the estimate is less than would be if it could be estimated from the total sample. Running the chains for more iterations will necessarily increase \(N_{ESS}\), but there is a practical demand on computational effort relative to marginal increases in precision. The recommendation of the Stan developers is to run enough iterations of the sampler to obtain an \(N_{ESS} >= 4*N_{chains}\). All models were run using 4 independent chains, so the minimally acceptable ESS is 400 (i.e, 4*100).
In the case that the model checks are appropriate, it is appropriate to examine the posterior distribution and begin inference based on the results. While there are more parameters estimated by the models, posterior summaries of the coefficients in each model are shown. The posterior is summarized as a density plot that reflects the probability distribution of the parameter based on integrating our prior knowledge and observed data. The density plot shows the 95% credible interval with the 80% credible interval shown as a shaded area. Unlike frequentist confidence intervals, these credible intervals can be interpreted as the probability of the parameter having a specific value. For example, if the 95% credible ranges from 0.50 to 1.00, then this means that there is a probability of 0.95 that the parameter has a value somewhere within this interval. This is in contrast to frequentist confidence intervals where the same interval would be interpreted as meaning that 95% of point estimates based on the same statistical test applied to an infinite number of random samples of the same population will be within this interval. Thus, where the credible interval directly summarizes the our beliefs about the parameter and our uncertainty about its true value, the confidence interval only reflects point estimates that we would expect to observe if the study and statistical methods were repeated an infinite number of times. Posterior predictive checks of the models are then also presented as they were for the final model in the corresponding manuscript. There is one additional model exploration plot provided in this section that has not been addressed before in this document: the conditional effects plot. As the predictors in these models are correlated and have their effects estimated on the logit scale, it can be challenging to look at the model estimates and understand the implication of these values in an intuitive manner. One way to address this is to visualize how the predicted outcome of the model changes as a function of each predictor while holding all other predictors constant (e.g., at their mean value). The resulting plot is the conditional effects plot. In the case of these models, this plot shows, for each predictor, what happens to the predicted probability of a correct response as the value of the predictor changes and all other model values are held constant. These plots are not realistic as it is not reasonable to assume that there exist words whose traits can vary only on one property at a time; however, they do provide a quick method of understanding the relative effect of each predictor by showing its linear trend as implied by the model. As a result, these plots should not be used for prediction or extrapolation in any regard; instead, if the goal is prediction of responses, then the entire model should be used, and extrapolation of these predictions to values not observed in this study should be avoided. These plots are simply to help contextualize the meaning of the effect estimate in the model.
Rasch Fixed Items
mcmc_plot(Rasch_inter, type = "nuts_divergence")
mcmc_plot(Rasch_inter, type = "nuts_treedepth")
mcmc_plot(Rasch_inter, type = "trace", variable = "b_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(Rasch_inter, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(Rasch_inter, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(Rasch_inter, type = "neff_hist", binwidth = 0.1)
The fixed items Rasch model demonstrated no evidence of estimation concerns that would raise concerns for the validity of the results. As a result, we can look at the results from the model overall.
mcmc_plot(Rasch_inter, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_Item", regex = TRUE)
pp_check(Rasch_inter, ndraws = 50, type = "bars")
pp_check(Rasch_inter, ndraws = 50, type = "bars_grouped", group = "Item")
pp_check(Rasch_inter, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
Even with no predictors and fewer parameters in the Rasch model, the model does very well in predicting responses. The density plot of the item easiness estimates demonstrates that most items are fairly easy. The following general model summary integrates basic model validity statistics and posterior summaries for additional parameters.
summary(Rasch_inter)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ 0 + Item + (1 | ID)
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept) 0.62 0.02 0.58 0.66 1.00 2712 3729
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## ItemButter1 1.32 0.07 1.17 1.46 1.00 6354 5699
## ItemArm1 -0.09 0.06 -0.21 0.03 1.00 6080 5708
## ItemShore1 -0.50 0.06 -0.62 -0.37 1.00 6764 5636
## ItemLetter1 -1.06 0.07 -1.20 -0.93 1.00 7192 5616
## ItemQueen1 -0.78 0.07 -0.91 -0.65 1.00 7200 5584
## ItemCabin1 -0.89 0.07 -1.02 -0.76 1.00 6397 5668
## ItemPole1 0.37 0.06 0.24 0.49 1.00 5281 4896
## ItemTicket1 -0.49 0.06 -0.62 -0.37 1.00 6757 5259
## ItemGrass1 0.79 0.07 0.66 0.92 1.00 6149 5198
## ItemEngine1 0.79 0.06 0.67 0.92 1.00 6451 5835
## ItemTicket2 1.26 0.07 1.12 1.40 1.00 6541 5871
## ItemCabin2 1.08 0.07 0.94 1.21 1.00 6313 5562
## ItemButter2 0.29 0.06 0.17 0.42 1.00 6613 5404
## ItemShore2 0.33 0.06 0.21 0.46 1.00 6690 5771
## ItemEngine2 0.93 0.07 0.80 1.06 1.00 6462 5718
## ItemArm2 0.35 0.06 0.23 0.47 1.00 5263 4942
## ItemQueen2 2.08 0.09 1.91 2.25 1.00 7259 5478
## ItemLetter2 0.85 0.07 0.72 0.98 1.00 6304 5246
## ItemPole2 2.38 0.10 2.19 2.59 1.00 7903 5077
## ItemGrass2 0.23 0.06 0.10 0.36 1.00 5965 5110
## ItemQueen3 1.41 0.07 1.27 1.56 1.00 7296 5380
## ItemGrass3 1.45 0.07 1.30 1.59 1.00 6759 6069
## ItemArm3 1.10 0.07 0.97 1.24 1.00 7319 5392
## ItemCabin3 1.74 0.08 1.59 1.90 1.00 6984 5378
## ItemPole3 1.65 0.08 1.49 1.81 1.00 5876 5353
## ItemShore3 0.67 0.06 0.54 0.80 1.00 5905 5024
## ItemButter3 1.33 0.07 1.19 1.48 1.00 7179 5516
## ItemEngine3 1.89 0.08 1.73 2.05 1.00 7327 5954
## ItemTicket3 0.97 0.07 0.84 1.11 1.00 6798 5470
## ItemLetter3 0.67 0.06 0.54 0.80 1.00 5181 5429
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).To help clarify the meaning of some major elements of the above output, consider the following guide: 1. “Estimate” refers to the average posterior value for the parameter, 2. “Est. Error” is the standard deviation of the posterior distribution, 3. “l-95% CI” is the lower bound of the 95% credible interval, 4. “u-95% CI” is the upper bound of the 95% credible interval, 5. “Rhat” is the \(\hat{R}\) value for that parameter (rounded to two decimal places), 6. “Bulk_ESS” is the effective sample size based on rank normalized draws and estimates the sampling efficiency of the mean of the posterior, and 7. “Tail_ESS” is the minimum of the effect sample sizes in the 5% and 95% quantiles.
2PL Fixed Items
mcmc_plot(TwoPL_inter, type = "nuts_divergence")
mcmc_plot(TwoPL_inter, type = "nuts_treedepth")
mcmc_plot(TwoPL_inter, type = "trace", variable = "b_beta_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_inter, type = "trace", variable = "b_logalpha_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_inter, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_inter, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(TwoPL_inter, type = "neff_hist", binwidth = 0.1)
The fixed item 2PL model demonstrated no evidence of estimation concerns that would raise concerns for the validity of the results. As a result, we can look at the results from the model overall.
mcmc_plot(TwoPL_inter, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)
mcmc_plot(TwoPL_inter, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)
pp_check(TwoPL_inter, ndraws = 50, type = "bars")
pp_check(TwoPL_inter, ndraws = 50, type = "bars_grouped", group = "Item")
pp_check(TwoPL_inter, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
Much like with the Rasch model, the 2PL fixed item model does very well in predicting responses. There are two parameters (one for difficulty/beta and one for discrimination/alpha) being estimated since the 2PL model is a non-linear model. Note that the model’s name for the discrimination parameter is “logalpha.” This name reflects the fact that the alpha parameter was log-transformed to ensure that it was constrained in estimation to positive values. The following general model summary integrates basic model validity statistics and posterior summaries for additional parameters. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.
summary(TwoPL_inter)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ beta + exp(logalpha) * theta
## theta ~ 0 + (1 | ID)
## beta ~ 0 + Item
## logalpha ~ 0 + Item
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept) 0.51 0.10 0.35 0.72 1.00 556 1166
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_ItemButter1 1.53 0.10 1.34 1.73 1.00 5640
## beta_ItemArm1 -0.10 0.08 -0.26 0.06 1.00 3975
## beta_ItemShore1 -0.56 0.08 -0.72 -0.42 1.00 6366
## beta_ItemLetter1 -1.05 0.07 -1.19 -0.91 1.00 10470
## beta_ItemQueen1 -0.81 0.07 -0.95 -0.67 1.00 7717
## beta_ItemCabin1 -0.87 0.07 -1.01 -0.74 1.00 9133
## beta_ItemPole1 0.34 0.06 0.22 0.46 1.00 14697
## beta_ItemTicket1 -0.48 0.06 -0.61 -0.36 1.00 10560
## beta_ItemGrass1 0.72 0.06 0.60 0.85 1.00 15464
## beta_ItemEngine1 0.73 0.06 0.61 0.85 1.00 13047
## beta_ItemTicket2 1.28 0.08 1.13 1.44 1.00 9791
## beta_ItemCabin2 1.25 0.09 1.07 1.43 1.00 5717
## beta_ItemButter2 0.32 0.07 0.19 0.46 1.00 6806
## beta_ItemShore2 0.31 0.06 0.19 0.43 1.00 11956
## beta_ItemEngine2 1.01 0.08 0.86 1.17 1.00 8287
## beta_ItemArm2 0.36 0.07 0.23 0.49 1.00 8820
## beta_ItemQueen2 1.95 0.09 1.78 2.13 1.00 12117
## beta_ItemLetter2 0.92 0.08 0.77 1.07 1.00 7447
## beta_ItemPole2 2.25 0.10 2.06 2.45 1.00 12961
## beta_ItemGrass2 0.21 0.06 0.10 0.33 1.00 12589
## beta_ItemQueen3 1.41 0.08 1.26 1.56 1.00 9991
## beta_ItemGrass3 1.66 0.10 1.47 1.87 1.00 7176
## beta_ItemArm3 1.20 0.08 1.04 1.36 1.00 7807
## beta_ItemCabin3 1.67 0.08 1.52 1.84 1.00 12315
## beta_ItemPole3 1.83 0.11 1.63 2.04 1.00 7351
## beta_ItemShore3 0.66 0.06 0.53 0.79 1.00 9841
## beta_ItemButter3 1.28 0.07 1.14 1.42 1.00 11118
## beta_ItemEngine3 1.96 0.10 1.77 2.17 1.00 10128
## beta_ItemTicket3 0.94 0.07 0.81 1.07 1.00 10910
## beta_ItemLetter3 0.64 0.06 0.51 0.77 1.00 11960
## logalpha_ItemButter1 0.84 0.21 0.41 1.24 1.00 692
## logalpha_ItemArm1 1.11 0.20 0.70 1.50 1.00 684
## logalpha_ItemShore1 0.82 0.21 0.40 1.21 1.00 664
## logalpha_ItemLetter1 0.06 0.24 -0.42 0.51 1.00 905
## logalpha_ItemQueen1 0.45 0.22 0.00 0.86 1.00 716
## logalpha_ItemCabin1 0.06 0.24 -0.43 0.51 1.00 874
## logalpha_ItemPole1 -0.98 0.38 -1.82 -0.33 1.00 2154
## logalpha_ItemTicket1 0.02 0.24 -0.45 0.48 1.00 859
## logalpha_ItemGrass1 -2.12 0.55 -3.31 -1.16 1.00 5486
## logalpha_ItemEngine1 -2.63 0.53 -3.75 -1.71 1.00 6066
## logalpha_ItemTicket2 0.33 0.23 -0.13 0.75 1.00 812
## logalpha_ItemCabin2 0.82 0.21 0.40 1.21 1.00 689
## logalpha_ItemButter2 0.58 0.21 0.15 0.98 1.00 718
## logalpha_ItemShore2 -0.48 0.30 -1.09 0.06 1.00 1287
## logalpha_ItemEngine2 0.63 0.21 0.22 1.04 1.00 693
## logalpha_ItemArm2 0.36 0.22 -0.09 0.77 1.00 739
## logalpha_ItemQueen2 -1.38 0.54 -2.59 -0.46 1.00 5657
## logalpha_ItemLetter2 0.58 0.21 0.15 0.99 1.00 691
## logalpha_ItemPole2 -1.20 0.55 -2.42 -0.27 1.00 4430
## logalpha_ItemGrass2 -0.83 0.36 -1.62 -0.20 1.00 2134
## logalpha_ItemQueen3 0.16 0.24 -0.34 0.60 1.00 903
## logalpha_ItemGrass3 0.81 0.22 0.38 1.22 1.00 677
## logalpha_ItemArm3 0.61 0.21 0.18 1.02 1.00 714
## logalpha_ItemCabin3 -0.28 0.31 -0.95 0.28 1.00 1353
## logalpha_ItemPole3 0.71 0.22 0.28 1.13 1.00 718
## logalpha_ItemShore3 0.09 0.24 -0.40 0.53 1.00 814
## logalpha_ItemButter3 -0.26 0.28 -0.84 0.27 1.00 1241
## logalpha_ItemEngine3 0.43 0.24 -0.05 0.88 1.00 895
## logalpha_ItemTicket3 -0.11 0.25 -0.63 0.36 1.00 1012
## logalpha_ItemLetter3 -0.20 0.26 -0.74 0.29 1.00 1053
## Tail_ESS
## beta_ItemButter1 5403
## beta_ItemArm1 5394
## beta_ItemShore1 5679
## beta_ItemLetter1 6140
## beta_ItemQueen1 5834
## beta_ItemCabin1 5303
## beta_ItemPole1 5038
## beta_ItemTicket1 5474
## beta_ItemGrass1 5777
## beta_ItemEngine1 5191
## beta_ItemTicket2 5960
## beta_ItemCabin2 5771
## beta_ItemButter2 4869
## beta_ItemShore2 4879
## beta_ItemEngine2 6284
## beta_ItemArm2 5991
## beta_ItemQueen2 5197
## beta_ItemLetter2 5662
## beta_ItemPole2 6019
## beta_ItemGrass2 5499
## beta_ItemQueen3 5249
## beta_ItemGrass3 5859
## beta_ItemArm3 5937
## beta_ItemCabin3 5919
## beta_ItemPole3 6353
## beta_ItemShore3 5977
## beta_ItemButter3 6230
## beta_ItemEngine3 6328
## beta_ItemTicket3 6488
## beta_ItemLetter3 5489
## logalpha_ItemButter1 1740
## logalpha_ItemArm1 1717
## logalpha_ItemShore1 1455
## logalpha_ItemLetter1 2594
## logalpha_ItemQueen1 1750
## logalpha_ItemCabin1 2248
## logalpha_ItemPole1 5034
## logalpha_ItemTicket1 2241
## logalpha_ItemGrass1 5571
## logalpha_ItemEngine1 5757
## logalpha_ItemTicket2 1998
## logalpha_ItemCabin2 1598
## logalpha_ItemButter2 1883
## logalpha_ItemShore2 2986
## logalpha_ItemEngine2 1901
## logalpha_ItemArm2 1973
## logalpha_ItemQueen2 5465
## logalpha_ItemLetter2 1801
## logalpha_ItemPole2 4753
## logalpha_ItemGrass2 4091
## logalpha_ItemQueen3 2300
## logalpha_ItemGrass3 1695
## logalpha_ItemArm3 1648
## logalpha_ItemCabin3 3445
## logalpha_ItemPole3 1795
## logalpha_ItemShore3 1938
## logalpha_ItemButter3 3003
## logalpha_ItemEngine3 2120
## logalpha_ItemTicket3 2563
## logalpha_ItemLetter3 3008
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Multidimensional 2PL Model
mcmc_plot(TwoPL_multi, type = "nuts_divergence")
mcmc_plot(TwoPL_multi, type = "nuts_treedepth")
mcmc_plot(TwoPL_multi, type = "trace", variable = "b_beta_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_multi, type = "trace", variable = "b_logalpha_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_multi, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_multi, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(TwoPL_multi, type = "neff_hist", binwidth = 0.1)
The 2PL model with a unique factor per trial shows no significant modeling concerns. Since there are no validity concerns, we can look at the results from the model overall.
mcmc_plot(TwoPL_multi, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)
mcmc_plot(TwoPL_multi, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)
pp_check(TwoPL_multi, ndraws = 50, type = "bars")
pp_check(TwoPL_multi, ndraws = 50, type = "bars_grouped", group = "Item")
pp_check(TwoPL_multi, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
Like the other 2PL models, this model estimates responses with high accuracy. The following general model summary integrates basic model validity statistics and posterior summaries for additional parameters. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.
summary(TwoPL_multi)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ beta + exp(logalpha) * theta
## theta ~ 0 + (-1 + Trial1 + Trial2 + Trial3 | ID)
## beta ~ 0 + Item
## logalpha ~ 0 + Item
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat
## sd(theta_Trial1) 0.44 0.15 0.22 0.79 1.00
## sd(theta_Trial2) 0.53 0.17 0.27 0.92 1.00
## sd(theta_Trial3) 0.67 0.21 0.35 1.15 1.00
## cor(theta_Trial1,theta_Trial2) 0.99 0.01 0.97 1.00 1.00
## cor(theta_Trial1,theta_Trial3) 0.99 0.01 0.96 1.00 1.00
## cor(theta_Trial2,theta_Trial3) 0.99 0.01 0.98 1.00 1.00
## Bulk_ESS Tail_ESS
## sd(theta_Trial1) 1018 1977
## sd(theta_Trial2) 1188 1757
## sd(theta_Trial3) 1171 2298
## cor(theta_Trial1,theta_Trial2) 1016 2169
## cor(theta_Trial1,theta_Trial3) 1158 2405
## cor(theta_Trial2,theta_Trial3) 3625 5411
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_ItemButter1 1.53 0.10 1.34 1.74 1.00 5482
## beta_ItemArm1 -0.10 0.08 -0.26 0.06 1.00 3456
## beta_ItemShore1 -0.57 0.08 -0.71 -0.42 1.00 4222
## beta_ItemLetter1 -1.05 0.07 -1.18 -0.91 1.00 8080
## beta_ItemQueen1 -0.81 0.07 -0.95 -0.67 1.00 7014
## beta_ItemCabin1 -0.87 0.07 -1.00 -0.74 1.00 7749
## beta_ItemPole1 0.34 0.06 0.22 0.46 1.00 11899
## beta_ItemTicket1 -0.48 0.06 -0.61 -0.36 1.00 7791
## beta_ItemGrass1 0.73 0.06 0.61 0.85 1.00 12302
## beta_ItemEngine1 0.73 0.06 0.61 0.85 1.00 11623
## beta_ItemTicket2 1.28 0.08 1.13 1.44 1.00 6665
## beta_ItemCabin2 1.25 0.09 1.07 1.43 1.00 5139
## beta_ItemButter2 0.32 0.07 0.19 0.46 1.00 5936
## beta_ItemShore2 0.31 0.06 0.20 0.43 1.00 8020
## beta_ItemEngine2 1.01 0.08 0.86 1.17 1.00 6205
## beta_ItemArm2 0.36 0.06 0.24 0.49 1.00 6621
## beta_ItemQueen2 1.95 0.09 1.79 2.12 1.00 9987
## beta_ItemLetter2 0.91 0.08 0.77 1.07 1.00 6395
## beta_ItemPole2 2.25 0.10 2.06 2.44 1.00 11506
## beta_ItemGrass2 0.21 0.06 0.10 0.33 1.00 10921
## beta_ItemQueen3 1.41 0.08 1.26 1.56 1.00 7601
## beta_ItemGrass3 1.67 0.10 1.48 1.88 1.00 5803
## beta_ItemArm3 1.20 0.08 1.04 1.36 1.00 6462
## beta_ItemCabin3 1.68 0.08 1.51 1.84 1.00 8818
## beta_ItemPole3 1.83 0.11 1.63 2.05 1.00 6337
## beta_ItemShore3 0.66 0.07 0.53 0.79 1.00 7665
## beta_ItemButter3 1.28 0.07 1.14 1.42 1.00 10672
## beta_ItemEngine3 1.96 0.10 1.76 2.16 1.00 7825
## beta_ItemTicket3 0.94 0.07 0.81 1.08 1.00 9432
## beta_ItemLetter3 0.64 0.06 0.52 0.76 1.00 9960
## logalpha_ItemButter1 1.03 0.34 0.37 1.68 1.00 1089
## logalpha_ItemArm1 1.30 0.33 0.65 1.95 1.00 1081
## logalpha_ItemShore1 1.00 0.33 0.34 1.65 1.00 1035
## logalpha_ItemLetter1 0.24 0.35 -0.45 0.91 1.00 1182
## logalpha_ItemQueen1 0.63 0.34 -0.04 1.28 1.00 1176
## logalpha_ItemCabin1 0.23 0.35 -0.47 0.91 1.00 1169
## logalpha_ItemPole1 -0.84 0.47 -1.84 -0.01 1.00 2309
## logalpha_ItemTicket1 0.20 0.35 -0.49 0.87 1.00 1228
## logalpha_ItemGrass1 -1.99 0.58 -3.25 -0.97 1.00 3697
## logalpha_ItemEngine1 -2.51 0.58 -3.70 -1.44 1.00 4255
## logalpha_ItemTicket2 0.32 0.34 -0.34 0.98 1.00 1323
## logalpha_ItemCabin2 0.81 0.33 0.16 1.46 1.00 1265
## logalpha_ItemButter2 0.57 0.33 -0.06 1.23 1.00 1309
## logalpha_ItemShore2 -0.50 0.38 -1.27 0.24 1.00 1697
## logalpha_ItemEngine2 0.62 0.33 -0.03 1.28 1.00 1257
## logalpha_ItemArm2 0.35 0.34 -0.31 1.01 1.00 1304
## logalpha_ItemQueen2 -1.38 0.57 -2.61 -0.39 1.00 3766
## logalpha_ItemLetter2 0.57 0.34 -0.08 1.24 1.00 1231
## logalpha_ItemPole2 -1.21 0.59 -2.47 -0.18 1.00 3363
## logalpha_ItemGrass2 -0.84 0.43 -1.75 -0.04 1.00 1899
## logalpha_ItemQueen3 -0.08 0.34 -0.74 0.58 1.00 1406
## logalpha_ItemGrass3 0.58 0.32 -0.05 1.20 1.00 1281
## logalpha_ItemArm3 0.38 0.32 -0.24 1.02 1.00 1256
## logalpha_ItemCabin3 -0.51 0.39 -1.30 0.22 1.00 1780
## logalpha_ItemPole3 0.48 0.32 -0.15 1.12 1.00 1268
## logalpha_ItemShore3 -0.15 0.33 -0.80 0.51 1.00 1390
## logalpha_ItemButter3 -0.49 0.37 -1.22 0.21 1.00 1660
## logalpha_ItemEngine3 0.19 0.33 -0.46 0.85 1.00 1361
## logalpha_ItemTicket3 -0.34 0.35 -1.03 0.34 1.00 1406
## logalpha_ItemLetter3 -0.44 0.35 -1.13 0.24 1.00 1474
## Tail_ESS
## beta_ItemButter1 5275
## beta_ItemArm1 5032
## beta_ItemShore1 5515
## beta_ItemLetter1 6064
## beta_ItemQueen1 6044
## beta_ItemCabin1 5888
## beta_ItemPole1 5168
## beta_ItemTicket1 5593
## beta_ItemGrass1 6109
## beta_ItemEngine1 5624
## beta_ItemTicket2 5443
## beta_ItemCabin2 5340
## beta_ItemButter2 5903
## beta_ItemShore2 5682
## beta_ItemEngine2 5683
## beta_ItemArm2 5630
## beta_ItemQueen2 5670
## beta_ItemLetter2 6121
## beta_ItemPole2 5996
## beta_ItemGrass2 5785
## beta_ItemQueen3 6006
## beta_ItemGrass3 5862
## beta_ItemArm3 5991
## beta_ItemCabin3 5832
## beta_ItemPole3 6117
## beta_ItemShore3 5955
## beta_ItemButter3 5742
## beta_ItemEngine3 5314
## beta_ItemTicket3 6410
## beta_ItemLetter3 6338
## logalpha_ItemButter1 2285
## logalpha_ItemArm1 2121
## logalpha_ItemShore1 2076
## logalpha_ItemLetter1 2162
## logalpha_ItemQueen1 2214
## logalpha_ItemCabin1 2281
## logalpha_ItemPole1 3893
## logalpha_ItemTicket1 2705
## logalpha_ItemGrass1 4997
## logalpha_ItemEngine1 5001
## logalpha_ItemTicket2 2375
## logalpha_ItemCabin2 2037
## logalpha_ItemButter2 2347
## logalpha_ItemShore2 2660
## logalpha_ItemEngine2 2069
## logalpha_ItemArm2 2035
## logalpha_ItemQueen2 4930
## logalpha_ItemLetter2 1991
## logalpha_ItemPole2 4908
## logalpha_ItemGrass2 3102
## logalpha_ItemQueen3 2761
## logalpha_ItemGrass3 2481
## logalpha_ItemArm3 2313
## logalpha_ItemCabin3 3285
## logalpha_ItemPole3 2301
## logalpha_ItemShore3 3015
## logalpha_ItemButter3 3267
## logalpha_ItemEngine3 2737
## logalpha_ItemTicket3 2989
## logalpha_ItemLetter3 2779
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Latent Change Model
mcmc_plot(TwoPL_chnge, type = "nuts_divergence")
mcmc_plot(TwoPL_chnge, type = "nuts_treedepth")
mcmc_plot(TwoPL_chnge, type = "trace", variable = "b_beta_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_chnge, type = "trace", variable = "b_logalpha_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_chnge, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_chnge, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(TwoPL_chnge, type = "neff_hist", binwidth = 0.1)
The 2PL model with factors for change over each trial shows no significant modeling concerns either. Since there are no validity concerns, we can look at the results from the model overall.
mcmc_plot(TwoPL_chnge, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)
mcmc_plot(TwoPL_chnge, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)
pp_check(TwoPL_chnge, ndraws = 50, type = "bars")
pp_check(TwoPL_chnge, ndraws = 50, type = "bars_grouped", group = "Item")
pp_check(TwoPL_chnge, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
Consistent with the other 2PL models so far, the model appears to predict the data generation process closely. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.
summary(TwoPL_chnge)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ beta + exp(logalpha) * theta
## theta ~ 0 + (-1 + Time1 + Time2 + Time3 | ID)
## beta ~ 0 + Item
## logalpha ~ 0 + Item
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## sd(theta_Time1) 0.52 0.10 0.35 0.75 1.01 759
## sd(theta_Time2) 0.05 0.05 0.00 0.19 1.01 1303
## sd(theta_Time3) 0.05 0.05 0.00 0.18 1.00 1992
## cor(theta_Time1,theta_Time2) 0.12 0.56 -0.90 0.96 1.00 4088
## cor(theta_Time1,theta_Time3) 0.11 0.57 -0.91 0.96 1.00 3986
## cor(theta_Time2,theta_Time3) 0.00 0.51 -0.88 0.90 1.00 5600
## Tail_ESS
## sd(theta_Time1) 1441
## sd(theta_Time2) 1700
## sd(theta_Time3) 1756
## cor(theta_Time1,theta_Time2) 2811
## cor(theta_Time1,theta_Time3) 2630
## cor(theta_Time2,theta_Time3) 5242
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_ItemButter1 1.53 0.10 1.33 1.74 1.00 4645
## beta_ItemArm1 -0.10 0.08 -0.25 0.06 1.00 3313
## beta_ItemShore1 -0.57 0.08 -0.72 -0.42 1.00 4745
## beta_ItemLetter1 -1.05 0.07 -1.18 -0.91 1.00 7191
## beta_ItemQueen1 -0.81 0.07 -0.96 -0.68 1.00 7443
## beta_ItemCabin1 -0.87 0.07 -1.01 -0.74 1.00 9514
## beta_ItemPole1 0.34 0.06 0.22 0.46 1.00 13174
## beta_ItemTicket1 -0.48 0.06 -0.61 -0.36 1.00 7845
## beta_ItemGrass1 0.73 0.06 0.61 0.85 1.00 14426
## beta_ItemEngine1 0.73 0.06 0.61 0.85 1.00 12021
## beta_ItemTicket2 1.28 0.08 1.13 1.44 1.00 8259
## beta_ItemCabin2 1.24 0.09 1.07 1.42 1.00 4965
## beta_ItemButter2 0.32 0.07 0.18 0.46 1.00 5622
## beta_ItemShore2 0.31 0.06 0.20 0.43 1.00 12137
## beta_ItemEngine2 1.01 0.08 0.86 1.17 1.00 6019
## beta_ItemArm2 0.36 0.07 0.23 0.49 1.00 6548
## beta_ItemQueen2 1.95 0.09 1.79 2.12 1.00 11104
## beta_ItemLetter2 0.92 0.08 0.77 1.07 1.00 6259
## beta_ItemPole2 2.25 0.10 2.06 2.45 1.00 11038
## beta_ItemGrass2 0.21 0.06 0.10 0.33 1.00 11943
## beta_ItemQueen3 1.41 0.08 1.26 1.56 1.00 8459
## beta_ItemGrass3 1.67 0.10 1.47 1.87 1.00 5593
## beta_ItemArm3 1.19 0.08 1.03 1.36 1.00 6328
## beta_ItemCabin3 1.68 0.08 1.52 1.84 1.00 10753
## beta_ItemPole3 1.83 0.10 1.63 2.04 1.00 6004
## beta_ItemShore3 0.66 0.07 0.53 0.79 1.00 8855
## beta_ItemButter3 1.28 0.07 1.14 1.42 1.00 9180
## beta_ItemEngine3 1.96 0.10 1.76 2.16 1.00 6610
## beta_ItemTicket3 0.94 0.07 0.81 1.07 1.00 8500
## beta_ItemLetter3 0.64 0.06 0.52 0.76 1.00 8680
## logalpha_ItemButter1 0.81 0.22 0.39 1.26 1.01 940
## logalpha_ItemArm1 1.08 0.21 0.68 1.50 1.01 876
## logalpha_ItemShore1 0.79 0.22 0.37 1.21 1.01 876
## logalpha_ItemLetter1 0.03 0.25 -0.46 0.51 1.00 1153
## logalpha_ItemQueen1 0.42 0.23 -0.02 0.85 1.01 966
## logalpha_ItemCabin1 0.02 0.24 -0.46 0.50 1.01 1193
## logalpha_ItemPole1 -1.02 0.40 -1.94 -0.34 1.00 2604
## logalpha_ItemTicket1 -0.01 0.24 -0.49 0.46 1.00 1110
## logalpha_ItemGrass1 -2.13 0.54 -3.32 -1.21 1.00 5225
## logalpha_ItemEngine1 -2.65 0.55 -3.80 -1.68 1.00 5299
## logalpha_ItemTicket2 0.27 0.23 -0.17 0.70 1.01 1099
## logalpha_ItemCabin2 0.76 0.21 0.34 1.16 1.01 890
## logalpha_ItemButter2 0.52 0.21 0.10 0.93 1.01 920
## logalpha_ItemShore2 -0.54 0.29 -1.16 0.00 1.00 1798
## logalpha_ItemEngine2 0.58 0.21 0.15 0.98 1.01 940
## logalpha_ItemArm2 0.30 0.22 -0.15 0.70 1.01 1016
## logalpha_ItemQueen2 -1.41 0.54 -2.66 -0.52 1.00 5032
## logalpha_ItemLetter2 0.52 0.21 0.10 0.93 1.01 934
## logalpha_ItemPole2 -1.23 0.53 -2.42 -0.35 1.00 5040
## logalpha_ItemGrass2 -0.88 0.35 -1.64 -0.29 1.00 2159
## logalpha_ItemQueen3 0.07 0.25 -0.44 0.54 1.01 1168
## logalpha_ItemGrass3 0.73 0.22 0.30 1.15 1.01 913
## logalpha_ItemArm3 0.53 0.22 0.09 0.95 1.01 952
## logalpha_ItemCabin3 -0.36 0.31 -1.02 0.21 1.01 1740
## logalpha_ItemPole3 0.63 0.22 0.20 1.06 1.01 954
## logalpha_ItemShore3 0.00 0.24 -0.48 0.47 1.01 1122
## logalpha_ItemButter3 -0.34 0.29 -0.93 0.19 1.01 1544
## logalpha_ItemEngine3 0.34 0.24 -0.15 0.79 1.01 1092
## logalpha_ItemTicket3 -0.19 0.26 -0.72 0.31 1.01 1355
## logalpha_ItemLetter3 -0.28 0.26 -0.83 0.21 1.01 1362
## Tail_ESS
## beta_ItemButter1 5414
## beta_ItemArm1 5023
## beta_ItemShore1 5481
## beta_ItemLetter1 5506
## beta_ItemQueen1 5691
## beta_ItemCabin1 6228
## beta_ItemPole1 5650
## beta_ItemTicket1 5974
## beta_ItemGrass1 6436
## beta_ItemEngine1 6461
## beta_ItemTicket2 5512
## beta_ItemCabin2 5753
## beta_ItemButter2 4808
## beta_ItemShore2 5526
## beta_ItemEngine2 5845
## beta_ItemArm2 5604
## beta_ItemQueen2 6067
## beta_ItemLetter2 5806
## beta_ItemPole2 6427
## beta_ItemGrass2 5513
## beta_ItemQueen3 6367
## beta_ItemGrass3 5832
## beta_ItemArm3 6040
## beta_ItemCabin3 6162
## beta_ItemPole3 5857
## beta_ItemShore3 5778
## beta_ItemButter3 6054
## beta_ItemEngine3 6254
## beta_ItemTicket3 5800
## beta_ItemLetter3 5448
## logalpha_ItemButter1 2225
## logalpha_ItemArm1 1823
## logalpha_ItemShore1 1910
## logalpha_ItemLetter1 2501
## logalpha_ItemQueen1 2104
## logalpha_ItemCabin1 2580
## logalpha_ItemPole1 4265
## logalpha_ItemTicket1 2472
## logalpha_ItemGrass1 5416
## logalpha_ItemEngine1 5200
## logalpha_ItemTicket2 2312
## logalpha_ItemCabin2 1952
## logalpha_ItemButter2 1967
## logalpha_ItemShore2 3391
## logalpha_ItemEngine2 2140
## logalpha_ItemArm2 2382
## logalpha_ItemQueen2 4633
## logalpha_ItemLetter2 2295
## logalpha_ItemPole2 5094
## logalpha_ItemGrass2 4639
## logalpha_ItemQueen3 2501
## logalpha_ItemGrass3 2258
## logalpha_ItemArm3 2362
## logalpha_ItemCabin3 3316
## logalpha_ItemPole3 2573
## logalpha_ItemShore3 2407
## logalpha_ItemButter3 3394
## logalpha_ItemEngine3 2586
## logalpha_ItemTicket3 3242
## logalpha_ItemLetter3 3007
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Latent Growth (Learning) Model
mcmc_plot(TwoPL_learn, type = "nuts_divergence")
mcmc_plot(TwoPL_learn, type = "nuts_treedepth")
mcmc_plot(TwoPL_learn, type = "trace", variable = "b_beta_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_learn, type = "trace", variable = "b_logalpha_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_learn, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_learn, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(TwoPL_learn, type = "neff_hist", binwidth = 0.1)
The 2PL model with a growth/learning factor also yields no estimation errors. Since there are no validity concerns for the model, we can look at the results from the model.
mcmc_plot(TwoPL_learn, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)
mcmc_plot(TwoPL_learn, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)
pp_check(TwoPL_learn, ndraws = 50, type = "bars")
pp_check(TwoPL_learn, ndraws = 50, type = "bars_grouped", group = "Item")
pp_check(TwoPL_learn, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
Continuing the trends of the other models, these posterior checks closely align with the observed data. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.
summary(TwoPL_learn)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ beta + exp(logalpha) * theta
## theta ~ 0 + (1 + Time.n | ID) + (-1 + Trial1 | ID) + (-1 + Trial2 | ID) + (-1 + Trial3 | ID)
## beta ~ 0 + Item
## logalpha ~ 0 + Item
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat
## sd(theta_Intercept) 0.55 0.12 0.32 0.81 1.01
## sd(theta_Time.n) 0.05 0.06 0.00 0.22 1.00
## sd(theta_Trial1) 0.03 0.03 0.00 0.10 1.00
## sd(theta_Trial2) 0.02 0.02 0.00 0.07 1.00
## sd(theta_Trial3) 0.03 0.03 0.00 0.10 1.00
## cor(theta_Intercept,theta_Time.n) 0.24 0.65 -0.96 0.99 1.00
## Bulk_ESS Tail_ESS
## sd(theta_Intercept) 729 1030
## sd(theta_Time.n) 900 1179
## sd(theta_Trial1) 1721 3222
## sd(theta_Trial2) 3787 4578
## sd(theta_Trial3) 2885 3484
## cor(theta_Intercept,theta_Time.n) 1943 2871
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_ItemButter1 1.53 0.10 1.34 1.74 1.00 6054
## beta_ItemArm1 -0.10 0.08 -0.26 0.06 1.00 4385
## beta_ItemShore1 -0.57 0.08 -0.72 -0.42 1.00 5129
## beta_ItemLetter1 -1.05 0.07 -1.18 -0.91 1.00 9892
## beta_ItemQueen1 -0.81 0.07 -0.95 -0.68 1.00 8400
## beta_ItemCabin1 -0.87 0.07 -1.01 -0.74 1.00 12392
## beta_ItemPole1 0.34 0.06 0.22 0.46 1.00 13341
## beta_ItemTicket1 -0.48 0.06 -0.61 -0.36 1.00 10550
## beta_ItemGrass1 0.72 0.06 0.60 0.85 1.00 15931
## beta_ItemEngine1 0.73 0.06 0.61 0.85 1.00 17977
## beta_ItemTicket2 1.28 0.08 1.13 1.44 1.00 9431
## beta_ItemCabin2 1.24 0.09 1.07 1.42 1.00 5939
## beta_ItemButter2 0.32 0.07 0.18 0.45 1.00 7000
## beta_ItemShore2 0.31 0.06 0.19 0.43 1.00 13969
## beta_ItemEngine2 1.01 0.08 0.86 1.17 1.00 7397
## beta_ItemArm2 0.36 0.07 0.23 0.49 1.00 8191
## beta_ItemQueen2 1.95 0.09 1.78 2.13 1.00 15205
## beta_ItemLetter2 0.91 0.07 0.77 1.06 1.00 6864
## beta_ItemPole2 2.25 0.10 2.06 2.45 1.00 16402
## beta_ItemGrass2 0.21 0.06 0.10 0.33 1.00 14702
## beta_ItemQueen3 1.41 0.08 1.25 1.57 1.00 10286
## beta_ItemGrass3 1.66 0.10 1.47 1.88 1.00 6546
## beta_ItemArm3 1.20 0.08 1.03 1.36 1.00 6909
## beta_ItemCabin3 1.68 0.08 1.52 1.84 1.00 11822
## beta_ItemPole3 1.83 0.11 1.63 2.04 1.00 6641
## beta_ItemShore3 0.66 0.07 0.53 0.79 1.00 10202
## beta_ItemButter3 1.28 0.07 1.14 1.42 1.00 11481
## beta_ItemEngine3 1.96 0.10 1.77 2.16 1.00 8372
## beta_ItemTicket3 0.94 0.07 0.81 1.07 1.00 11121
## beta_ItemLetter3 0.64 0.06 0.52 0.76 1.00 11140
## logalpha_ItemButter1 0.77 0.25 0.31 1.31 1.00 865
## logalpha_ItemArm1 1.04 0.25 0.60 1.57 1.01 819
## logalpha_ItemShore1 0.75 0.25 0.29 1.27 1.00 836
## logalpha_ItemLetter1 -0.01 0.27 -0.54 0.55 1.00 1049
## logalpha_ItemQueen1 0.38 0.25 -0.10 0.92 1.00 899
## logalpha_ItemCabin1 -0.01 0.27 -0.54 0.54 1.00 996
## logalpha_ItemPole1 -1.05 0.41 -1.95 -0.33 1.00 2110
## logalpha_ItemTicket1 -0.05 0.27 -0.57 0.52 1.00 980
## logalpha_ItemGrass1 -2.17 0.54 -3.33 -1.23 1.00 4892
## logalpha_ItemEngine1 -2.70 0.55 -3.87 -1.69 1.00 5602
## logalpha_ItemTicket2 0.20 0.23 -0.27 0.65 1.00 942
## logalpha_ItemCabin2 0.69 0.21 0.28 1.11 1.00 839
## logalpha_ItemButter2 0.45 0.21 0.04 0.88 1.00 846
## logalpha_ItemShore2 -0.60 0.29 -1.23 -0.07 1.00 1507
## logalpha_ItemEngine2 0.51 0.21 0.09 0.92 1.00 858
## logalpha_ItemArm2 0.23 0.22 -0.19 0.65 1.00 892
## logalpha_ItemQueen2 -1.46 0.54 -2.70 -0.56 1.00 4673
## logalpha_ItemLetter2 0.45 0.22 0.03 0.87 1.00 854
## logalpha_ItemPole2 -1.28 0.52 -2.49 -0.42 1.00 4148
## logalpha_ItemGrass2 -0.94 0.35 -1.73 -0.34 1.00 2307
## logalpha_ItemQueen3 -0.01 0.26 -0.55 0.48 1.00 1021
## logalpha_ItemGrass3 0.64 0.24 0.16 1.10 1.00 774
## logalpha_ItemArm3 0.44 0.24 -0.05 0.91 1.00 833
## logalpha_ItemCabin3 -0.44 0.33 -1.15 0.14 1.00 1490
## logalpha_ItemPole3 0.55 0.24 0.06 1.00 1.00 790
## logalpha_ItemShore3 -0.08 0.26 -0.59 0.40 1.00 934
## logalpha_ItemButter3 -0.42 0.30 -1.05 0.14 1.00 1241
## logalpha_ItemEngine3 0.25 0.26 -0.29 0.74 1.00 957
## logalpha_ItemTicket3 -0.27 0.28 -0.84 0.25 1.00 1053
## logalpha_ItemLetter3 -0.37 0.28 -0.94 0.16 1.00 1196
## Tail_ESS
## beta_ItemButter1 5753
## beta_ItemArm1 5361
## beta_ItemShore1 5686
## beta_ItemLetter1 6420
## beta_ItemQueen1 6089
## beta_ItemCabin1 5795
## beta_ItemPole1 5727
## beta_ItemTicket1 5657
## beta_ItemGrass1 5732
## beta_ItemEngine1 5627
## beta_ItemTicket2 5657
## beta_ItemCabin2 5659
## beta_ItemButter2 5289
## beta_ItemShore2 5468
## beta_ItemEngine2 5708
## beta_ItemArm2 5316
## beta_ItemQueen2 5656
## beta_ItemLetter2 6187
## beta_ItemPole2 5756
## beta_ItemGrass2 5306
## beta_ItemQueen3 5577
## beta_ItemGrass3 6185
## beta_ItemArm3 6322
## beta_ItemCabin3 6117
## beta_ItemPole3 6204
## beta_ItemShore3 5276
## beta_ItemButter3 5670
## beta_ItemEngine3 5910
## beta_ItemTicket3 5275
## beta_ItemLetter3 5728
## logalpha_ItemButter1 1317
## logalpha_ItemArm1 1186
## logalpha_ItemShore1 1244
## logalpha_ItemLetter1 1671
## logalpha_ItemQueen1 1279
## logalpha_ItemCabin1 1553
## logalpha_ItemPole1 2657
## logalpha_ItemTicket1 1435
## logalpha_ItemGrass1 5177
## logalpha_ItemEngine1 5197
## logalpha_ItemTicket2 2128
## logalpha_ItemCabin2 1821
## logalpha_ItemButter2 1867
## logalpha_ItemShore2 3244
## logalpha_ItemEngine2 1883
## logalpha_ItemArm2 2364
## logalpha_ItemQueen2 5111
## logalpha_ItemLetter2 1937
## logalpha_ItemPole2 4795
## logalpha_ItemGrass2 3924
## logalpha_ItemQueen3 2516
## logalpha_ItemGrass3 2006
## logalpha_ItemArm3 2051
## logalpha_ItemCabin3 2853
## logalpha_ItemPole3 1828
## logalpha_ItemShore3 2556
## logalpha_ItemButter3 2939
## logalpha_ItemEngine3 2295
## logalpha_ItemTicket3 2586
## logalpha_ItemLetter3 2813
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Local Dependency Model
mcmc_plot(TwoPL_depmd, type = "nuts_divergence")
mcmc_plot(TwoPL_depmd, type = "nuts_treedepth")
mcmc_plot(TwoPL_depmd, type = "trace", variable = "b_beta_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_depmd, type = "trace", variable = "b_logalpha_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_depmd, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_depmd, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(TwoPL_depmd, type = "neff_hist", binwidth = 0.1)
The local dependency effect on the 2PL model fit without any modeling concerns. Given the lack of validity issues, the results from the model overall are shown next.
mcmc_plot(TwoPL_depmd, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)
mcmc_plot(TwoPL_depmd, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)
pp_check(TwoPL_depmd, ndraws = 50, type = "bars")
pp_check(TwoPL_depmd, ndraws = 50, type = "bars_grouped", group = "Item")
pp_check(TwoPL_depmd, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
As with the other models, the predictive checks closely align to the data. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.
summary(TwoPL_depmd)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ beta + exp(logalpha) * theta
## theta ~ 0 + (1 | ID)
## beta ~ 0 + LocDep + Item
## logalpha ~ 0 + LocDep + Item
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept) 0.41 0.08 0.27 0.58 1.00 935 1800
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_LocDep 0.60 0.04 0.53 0.68 1.00 5052
## beta_ItemButter1 1.52 0.10 1.34 1.72 1.00 6091
## beta_ItemArm1 -0.10 0.08 -0.26 0.06 1.00 5202
## beta_ItemShore1 -0.56 0.08 -0.71 -0.41 1.00 6770
## beta_ItemLetter1 -1.04 0.07 -1.18 -0.91 1.00 9319
## beta_ItemQueen1 -0.80 0.07 -0.94 -0.67 1.00 7982
## beta_ItemCabin1 -0.86 0.07 -0.99 -0.73 1.00 9718
## beta_ItemPole1 0.34 0.06 0.23 0.45 1.00 11492
## beta_ItemTicket1 -0.48 0.06 -0.60 -0.36 1.00 10765
## beta_ItemGrass1 0.73 0.06 0.61 0.85 1.00 11512
## beta_ItemEngine1 0.73 0.06 0.61 0.85 1.00 12922
## beta_ItemTicket2 0.79 0.08 0.64 0.95 1.00 7419
## beta_ItemCabin2 0.92 0.09 0.75 1.10 1.00 6568
## beta_ItemButter2 0.07 0.07 -0.06 0.20 1.00 6842
## beta_ItemShore2 0.15 0.06 0.04 0.27 1.00 10930
## beta_ItemEngine2 0.81 0.08 0.66 0.97 1.00 8139
## beta_ItemArm2 0.17 0.06 0.05 0.30 1.00 8543
## beta_ItemQueen2 1.63 0.09 1.46 1.81 1.00 9460
## beta_ItemLetter2 0.69 0.08 0.54 0.84 1.00 7856
## beta_ItemPole2 1.88 0.10 1.68 2.08 1.00 11310
## beta_ItemGrass2 -0.19 0.06 -0.32 -0.07 1.00 9334
## beta_ItemQueen3 0.99 0.08 0.83 1.16 1.00 7728
## beta_ItemGrass3 1.33 0.11 1.13 1.55 1.00 5763
## beta_ItemArm3 0.73 0.09 0.57 0.91 1.00 6118
## beta_ItemCabin3 1.34 0.08 1.18 1.51 1.00 9223
## beta_ItemPole3 1.55 0.11 1.35 1.77 1.00 7472
## beta_ItemShore3 0.20 0.07 0.07 0.34 1.00 7356
## beta_ItemButter3 0.85 0.08 0.71 1.00 1.00 8224
## beta_ItemEngine3 1.65 0.11 1.45 1.87 1.00 8041
## beta_ItemTicket3 0.42 0.07 0.28 0.57 1.00 7823
## beta_ItemLetter3 0.10 0.07 -0.04 0.24 1.00 7396
## logalpha_LocDep 0.21 0.08 0.06 0.36 1.00 6010
## logalpha_ItemButter1 1.06 0.22 0.62 1.49 1.00 1246
## logalpha_ItemArm1 1.33 0.22 0.90 1.76 1.00 1124
## logalpha_ItemShore1 1.02 0.22 0.59 1.44 1.00 1182
## logalpha_ItemLetter1 0.24 0.25 -0.27 0.72 1.00 1504
## logalpha_ItemQueen1 0.64 0.23 0.20 1.08 1.00 1333
## logalpha_ItemCabin1 0.18 0.26 -0.34 0.67 1.00 1451
## logalpha_ItemPole1 -0.88 0.44 -1.88 -0.16 1.00 3640
## logalpha_ItemTicket1 0.20 0.25 -0.31 0.69 1.00 1392
## logalpha_ItemGrass1 -2.00 0.58 -3.28 -1.00 1.00 5679
## logalpha_ItemEngine1 -2.50 0.56 -3.70 -1.49 1.00 6114
## logalpha_ItemTicket2 0.10 0.27 -0.44 0.60 1.00 1721
## logalpha_ItemCabin2 0.75 0.23 0.29 1.21 1.00 1340
## logalpha_ItemButter2 0.50 0.23 0.03 0.94 1.00 1331
## logalpha_ItemShore2 -0.71 0.38 -1.56 -0.04 1.00 2864
## logalpha_ItemEngine2 0.65 0.23 0.20 1.09 1.00 1316
## logalpha_ItemArm2 0.34 0.23 -0.13 0.80 1.00 1326
## logalpha_ItemQueen2 -1.43 0.57 -2.64 -0.46 1.00 6123
## logalpha_ItemLetter2 0.67 0.23 0.23 1.11 1.00 1134
## logalpha_ItemPole2 -1.17 0.57 -2.45 -0.21 1.00 5408
## logalpha_ItemGrass2 -0.75 0.38 -1.58 -0.10 1.00 2799
## logalpha_ItemQueen3 -0.01 0.29 -0.60 0.52 1.00 2059
## logalpha_ItemGrass3 0.87 0.23 0.43 1.32 1.00 1218
## logalpha_ItemArm3 0.61 0.23 0.16 1.06 1.00 1267
## logalpha_ItemCabin3 -0.76 0.47 -1.84 0.01 1.00 3541
## logalpha_ItemPole3 0.86 0.22 0.43 1.30 1.00 1225
## logalpha_ItemShore3 -0.15 0.28 -0.75 0.37 1.00 1672
## logalpha_ItemButter3 -0.63 0.40 -1.53 0.05 1.00 2712
## logalpha_ItemEngine3 0.54 0.25 0.05 1.01 1.00 1338
## logalpha_ItemTicket3 -0.05 0.27 -0.59 0.45 1.00 1744
## logalpha_ItemLetter3 -0.21 0.28 -0.78 0.31 1.00 1738
## Tail_ESS
## beta_LocDep 6083
## beta_ItemButter1 5669
## beta_ItemArm1 5179
## beta_ItemShore1 5106
## beta_ItemLetter1 6352
## beta_ItemQueen1 5888
## beta_ItemCabin1 6218
## beta_ItemPole1 6070
## beta_ItemTicket1 5935
## beta_ItemGrass1 5620
## beta_ItemEngine1 5695
## beta_ItemTicket2 5932
## beta_ItemCabin2 5477
## beta_ItemButter2 5276
## beta_ItemShore2 5271
## beta_ItemEngine2 5757
## beta_ItemArm2 5844
## beta_ItemQueen2 5528
## beta_ItemLetter2 5739
## beta_ItemPole2 5564
## beta_ItemGrass2 5824
## beta_ItemQueen3 5210
## beta_ItemGrass3 5420
## beta_ItemArm3 6124
## beta_ItemCabin3 6130
## beta_ItemPole3 6112
## beta_ItemShore3 6219
## beta_ItemButter3 5757
## beta_ItemEngine3 5856
## beta_ItemTicket3 5887
## beta_ItemLetter3 5773
## logalpha_LocDep 6098
## logalpha_ItemButter1 2585
## logalpha_ItemArm1 2320
## logalpha_ItemShore1 2367
## logalpha_ItemLetter1 3319
## logalpha_ItemQueen1 2818
## logalpha_ItemCabin1 3219
## logalpha_ItemPole1 4464
## logalpha_ItemTicket1 3203
## logalpha_ItemGrass1 3779
## logalpha_ItemEngine1 5411
## logalpha_ItemTicket2 3662
## logalpha_ItemCabin2 2742
## logalpha_ItemButter2 2979
## logalpha_ItemShore2 4071
## logalpha_ItemEngine2 2735
## logalpha_ItemArm2 2678
## logalpha_ItemQueen2 5345
## logalpha_ItemLetter2 2673
## logalpha_ItemPole2 4571
## logalpha_ItemGrass2 4210
## logalpha_ItemQueen3 3764
## logalpha_ItemGrass3 2440
## logalpha_ItemArm3 2617
## logalpha_ItemCabin3 4601
## logalpha_ItemPole3 2642
## logalpha_ItemShore3 2949
## logalpha_ItemButter3 4151
## logalpha_ItemEngine3 2649
## logalpha_ItemTicket3 3657
## logalpha_ItemLetter3 3641
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Local Dependency by Item Model
mcmc_plot(TwoPL_depun, type = "nuts_divergence")
mcmc_plot(TwoPL_depun, type = "nuts_treedepth")## Warning: Groups with fewer than two data points have been dropped.
## Warning: Groups with fewer than two data points have been dropped.
mcmc_plot(TwoPL_depun, type = "trace", variable = "b_beta_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_depun, type = "trace", variable = "b_logalpha_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_depun, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_depun, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(TwoPL_depun, type = "neff_hist", binwidth = 0.1)
As with the model with a uniform local dependency effect, the model with a unique dependency effect per item demonstrated no estimation errors, so next is shown the results from the model overall.
mcmc_plot(TwoPL_depun, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)
mcmc_plot(TwoPL_depun, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)
pp_check(TwoPL_depun, ndraws = 50, type = "bars")
pp_check(TwoPL_depun, ndraws = 50, type = "bars_grouped", group = "Item")
pp_check(TwoPL_depun, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
These posterior predictive checks also strongly align with the observed data. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.
summary(TwoPL_depun)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ beta + exp(logalpha) * theta
## theta ~ 0 + (1 | ID)
## beta ~ 0 + DepButter + DepArm + DepShore + DepLetter + DepQueen + DepCabin + DepPole + DepTicket + DepGrass + DepEngine + Item
## logalpha ~ 0 + DepButter + DepArm + DepShore + DepLetter + DepQueen + DepCabin + DepPole + DepTicket + DepGrass + DepEngine + Item
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept) 0.41 0.08 0.27 0.60 1.00 1656 3063
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_DepButter 0.42 0.15 0.13 0.70 1.00 6081
## beta_DepArm 0.61 0.14 0.33 0.88 1.00 6854
## beta_DepShore 0.72 0.12 0.49 0.94 1.00 8988
## beta_DepLetter 0.97 0.11 0.76 1.19 1.00 11481
## beta_DepQueen 0.50 0.13 0.24 0.76 1.00 9653
## beta_DepCabin 0.45 0.11 0.24 0.66 1.00 11129
## beta_DepPole 0.60 0.12 0.37 0.83 1.00 11237
## beta_DepTicket 0.60 0.13 0.36 0.85 1.00 11049
## beta_DepGrass 0.64 0.14 0.37 0.90 1.00 11050
## beta_DepEngine 0.55 0.11 0.33 0.76 1.00 10940
## beta_ItemButter1 1.53 0.10 1.35 1.74 1.00 9297
## beta_ItemArm1 -0.10 0.08 -0.27 0.06 1.00 7705
## beta_ItemShore1 -0.56 0.08 -0.71 -0.42 1.00 8562
## beta_ItemLetter1 -1.04 0.07 -1.17 -0.90 1.00 13880
## beta_ItemQueen1 -0.81 0.07 -0.95 -0.67 1.00 12415
## beta_ItemCabin1 -0.87 0.07 -1.00 -0.73 1.00 14653
## beta_ItemPole1 0.34 0.06 0.22 0.45 1.00 16442
## beta_ItemTicket1 -0.48 0.06 -0.60 -0.36 1.00 13703
## beta_ItemGrass1 0.73 0.06 0.60 0.85 1.00 16950
## beta_ItemEngine1 0.73 0.06 0.61 0.85 1.00 17010
## beta_ItemTicket2 0.95 0.16 0.65 1.27 1.00 5671
## beta_ItemCabin2 0.88 0.11 0.66 1.11 1.00 6654
## beta_ItemButter2 0.02 0.08 -0.13 0.18 1.00 8007
## beta_ItemShore2 0.07 0.06 -0.05 0.20 1.00 14322
## beta_ItemEngine2 0.85 0.09 0.67 1.03 1.00 8126
## beta_ItemArm2 0.22 0.07 0.08 0.36 1.00 9590
## beta_ItemQueen2 1.64 0.10 1.44 1.84 1.00 12186
## beta_ItemLetter2 0.70 0.09 0.53 0.88 1.00 8609
## beta_ItemPole2 1.86 0.12 1.62 2.11 1.00 9995
## beta_ItemGrass2 -0.16 0.09 -0.34 0.03 1.00 10008
## beta_ItemQueen3 1.12 0.13 0.87 1.39 1.00 5995
## beta_ItemGrass3 1.32 0.12 1.09 1.57 1.00 7801
## beta_ItemArm3 0.64 0.12 0.42 0.88 1.00 7561
## beta_ItemCabin3 1.19 0.09 1.01 1.38 1.00 12009
## beta_ItemPole3 1.59 0.12 1.36 1.83 1.00 9393
## beta_ItemShore3 0.32 0.10 0.12 0.52 1.00 10480
## beta_ItemButter3 0.85 0.11 0.65 1.07 1.00 11056
## beta_ItemEngine3 1.65 0.12 1.42 1.89 1.00 10334
## beta_ItemTicket3 0.40 0.13 0.14 0.66 1.00 10781
## beta_ItemLetter3 0.15 0.12 -0.08 0.37 1.00 9981
## logalpha_DepButter 0.03 0.33 -0.56 0.73 1.00 5525
## logalpha_DepArm 0.45 0.19 0.08 0.83 1.00 6001
## logalpha_DepShore 0.24 0.19 -0.13 0.63 1.00 6948
## logalpha_DepLetter -0.58 0.67 -2.04 0.58 1.00 11320
## logalpha_DepQueen 0.13 0.17 -0.21 0.47 1.00 8643
## logalpha_DepCabin 0.07 0.22 -0.37 0.49 1.00 9773
## logalpha_DepPole 0.34 0.54 -0.81 1.33 1.00 9536
## logalpha_DepTicket 0.10 0.19 -0.28 0.46 1.00 9028
## logalpha_DepGrass 0.29 0.40 -0.43 1.14 1.00 9816
## logalpha_DepEngine -0.09 0.38 -0.79 0.71 1.00 8309
## logalpha_ItemButter1 1.06 0.22 0.63 1.50 1.00 1987
## logalpha_ItemArm1 1.33 0.22 0.89 1.75 1.00 1940
## logalpha_ItemShore1 0.99 0.22 0.55 1.42 1.00 2050
## logalpha_ItemLetter1 0.20 0.26 -0.33 0.68 1.00 2674
## logalpha_ItemQueen1 0.64 0.23 0.20 1.07 1.00 2251
## logalpha_ItemCabin1 0.20 0.25 -0.33 0.69 1.00 2641
## logalpha_ItemPole1 -0.91 0.43 -1.90 -0.18 1.00 5389
## logalpha_ItemTicket1 0.19 0.25 -0.32 0.67 1.00 2197
## logalpha_ItemGrass1 -2.01 0.57 -3.24 -1.02 1.00 9930
## logalpha_ItemEngine1 -2.51 0.55 -3.69 -1.51 1.00 8689
## logalpha_ItemTicket2 0.29 0.38 -0.52 0.96 1.00 3783
## logalpha_ItemCabin2 0.61 0.27 0.07 1.13 1.00 2511
## logalpha_ItemButter2 0.40 0.27 -0.16 0.91 1.00 2582
## logalpha_ItemShore2 -0.85 0.46 -1.90 -0.09 1.00 5712
## logalpha_ItemEngine2 0.69 0.24 0.21 1.16 1.00 2535
## logalpha_ItemArm2 0.43 0.26 -0.09 0.91 1.00 2527
## logalpha_ItemQueen2 -1.52 0.58 -2.76 -0.52 1.00 8274
## logalpha_ItemLetter2 0.70 0.24 0.22 1.17 1.00 2229
## logalpha_ItemPole2 -1.29 0.62 -2.60 -0.18 1.00 6796
## logalpha_ItemGrass2 -0.53 0.48 -1.57 0.30 1.00 4745
## logalpha_ItemQueen3 0.13 0.32 -0.54 0.73 1.00 3473
## logalpha_ItemGrass3 0.74 0.25 0.25 1.23 1.00 2349
## logalpha_ItemArm3 0.55 0.26 0.02 1.04 1.00 2336
## logalpha_ItemCabin3 -0.96 0.56 -2.22 -0.02 1.00 6693
## logalpha_ItemPole3 0.86 0.24 0.39 1.32 1.00 2307
## logalpha_ItemShore3 -0.01 0.30 -0.62 0.56 1.00 3414
## logalpha_ItemButter3 -0.78 0.47 -1.77 0.04 1.00 5995
## logalpha_ItemEngine3 0.56 0.26 0.05 1.07 1.00 2321
## logalpha_ItemTicket3 -0.14 0.40 -0.99 0.60 1.00 5791
## logalpha_ItemLetter3 -0.02 0.38 -0.82 0.68 1.00 5003
## Tail_ESS
## beta_DepButter 6305
## beta_DepArm 6434
## beta_DepShore 6605
## beta_DepLetter 6392
## beta_DepQueen 6960
## beta_DepCabin 6571
## beta_DepPole 6405
## beta_DepTicket 6888
## beta_DepGrass 5483
## beta_DepEngine 6014
## beta_ItemButter1 6675
## beta_ItemArm1 6290
## beta_ItemShore1 6256
## beta_ItemLetter1 6005
## beta_ItemQueen1 6313
## beta_ItemCabin1 6020
## beta_ItemPole1 5714
## beta_ItemTicket1 5669
## beta_ItemGrass1 5472
## beta_ItemEngine1 5420
## beta_ItemTicket2 6668
## beta_ItemCabin2 6647
## beta_ItemButter2 6183
## beta_ItemShore2 6317
## beta_ItemEngine2 6954
## beta_ItemArm2 6496
## beta_ItemQueen2 6594
## beta_ItemLetter2 6312
## beta_ItemPole2 6735
## beta_ItemGrass2 6506
## beta_ItemQueen3 6031
## beta_ItemGrass3 6436
## beta_ItemArm3 6478
## beta_ItemCabin3 6690
## beta_ItemPole3 6721
## beta_ItemShore3 6785
## beta_ItemButter3 6243
## beta_ItemEngine3 6963
## beta_ItemTicket3 6561
## beta_ItemLetter3 6386
## logalpha_DepButter 5674
## logalpha_DepArm 6063
## logalpha_DepShore 5732
## logalpha_DepLetter 6450
## logalpha_DepQueen 6350
## logalpha_DepCabin 6439
## logalpha_DepPole 5927
## logalpha_DepTicket 6617
## logalpha_DepGrass 6399
## logalpha_DepEngine 6109
## logalpha_ItemButter1 3694
## logalpha_ItemArm1 3262
## logalpha_ItemShore1 3709
## logalpha_ItemLetter1 4215
## logalpha_ItemQueen1 4204
## logalpha_ItemCabin1 4290
## logalpha_ItemPole1 5529
## logalpha_ItemTicket1 4127
## logalpha_ItemGrass1 5348
## logalpha_ItemEngine1 5090
## logalpha_ItemTicket2 4954
## logalpha_ItemCabin2 3596
## logalpha_ItemButter2 4209
## logalpha_ItemShore2 5933
## logalpha_ItemEngine2 4592
## logalpha_ItemArm2 4178
## logalpha_ItemQueen2 6244
## logalpha_ItemLetter2 4316
## logalpha_ItemPole2 6412
## logalpha_ItemGrass2 5358
## logalpha_ItemQueen3 5186
## logalpha_ItemGrass3 4325
## logalpha_ItemArm3 4227
## logalpha_ItemCabin3 5820
## logalpha_ItemPole3 4163
## logalpha_ItemShore3 5113
## logalpha_ItemButter3 6520
## logalpha_ItemEngine3 4258
## logalpha_ItemTicket3 5683
## logalpha_ItemLetter3 6095
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Local Dependency by Trial Model
mcmc_plot(TwoPL_deptr, type = "nuts_divergence")
mcmc_plot(TwoPL_deptr, type = "nuts_treedepth")
mcmc_plot(TwoPL_deptr, type = "trace", variable = "b_beta_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_deptr, type = "trace", variable = "b_logalpha_Item", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_deptr, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_deptr, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(TwoPL_deptr, type = "neff_hist", binwidth = 0.1)
Decomposing the dependency matrix into unique trial effects, rather than unique item effects, also does not cause any modeling validity concerns. As with the other models, the overall results are shown next.
mcmc_plot(TwoPL_deptr, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)
mcmc_plot(TwoPL_deptr, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)
pp_check(TwoPL_deptr, ndraws = 50, type = "bars")
pp_check(TwoPL_deptr, ndraws = 50, type = "bars_grouped", group = "Item")
pp_check(TwoPL_deptr, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
Posterior predictive checks are consistent with the other models in that the model predictions and observed data closely align. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.
summary(TwoPL_deptr)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ beta + exp(logalpha) * theta
## theta ~ 0 + (1 | ID)
## beta ~ 0 + LocDep.1 + LocDep.2 + Item
## logalpha ~ 0 + LocDep.1 + LocDep.2 + Item
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept) 0.47 0.09 0.31 0.67 1.00 781 1188
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_LocDep.1 1.03 0.05 0.93 1.13 1.00 5496
## beta_LocDep.2 0.12 0.06 0.00 0.24 1.00 2727
## beta_ItemButter1 1.51 0.10 1.33 1.72 1.00 6278
## beta_ItemArm1 -0.10 0.08 -0.26 0.06 1.00 5321
## beta_ItemShore1 -0.56 0.08 -0.71 -0.41 1.00 6364
## beta_ItemLetter1 -1.05 0.07 -1.19 -0.91 1.00 10297
## beta_ItemQueen1 -0.80 0.07 -0.94 -0.67 1.00 7502
## beta_ItemCabin1 -0.87 0.07 -1.00 -0.74 1.00 9728
## beta_ItemPole1 0.34 0.06 0.22 0.46 1.00 11433
## beta_ItemTicket1 -0.48 0.06 -0.60 -0.36 1.00 10139
## beta_ItemGrass1 0.73 0.06 0.61 0.85 1.00 11444
## beta_ItemEngine1 0.73 0.06 0.61 0.85 1.00 11609
## beta_ItemTicket2 0.51 0.08 0.35 0.68 1.00 6772
## beta_ItemCabin2 0.80 0.09 0.63 0.98 1.00 5746
## beta_ItemButter2 -0.01 0.07 -0.15 0.13 1.00 6227
## beta_ItemShore2 0.07 0.06 -0.05 0.19 1.00 9760
## beta_ItemEngine2 0.76 0.08 0.60 0.92 1.00 6575
## beta_ItemArm2 0.09 0.07 -0.05 0.23 1.00 7097
## beta_ItemQueen2 1.45 0.09 1.27 1.64 1.00 9284
## beta_ItemLetter2 0.58 0.08 0.43 0.73 1.00 6801
## beta_ItemPole2 1.65 0.10 1.46 1.86 1.00 9731
## beta_ItemGrass2 -0.49 0.07 -0.63 -0.36 1.00 7036
## beta_ItemQueen3 1.30 0.09 1.13 1.48 1.00 4605
## beta_ItemGrass3 1.56 0.11 1.35 1.78 1.00 4352
## beta_ItemArm3 1.08 0.09 0.90 1.27 1.00 4051
## beta_ItemCabin3 1.59 0.09 1.42 1.76 1.00 5495
## beta_ItemPole3 1.77 0.11 1.56 2.00 1.00 5384
## beta_ItemShore3 0.55 0.08 0.40 0.70 1.00 4006
## beta_ItemButter3 1.17 0.08 1.01 1.34 1.00 4982
## beta_ItemEngine3 1.87 0.11 1.67 2.08 1.00 6741
## beta_ItemTicket3 0.83 0.09 0.67 1.00 1.00 4622
## beta_ItemLetter3 0.53 0.08 0.36 0.69 1.00 4170
## logalpha_LocDep.1 -0.57 0.17 -0.94 -0.26 1.00 5977
## logalpha_LocDep.2 0.28 0.11 0.07 0.49 1.00 3280
## logalpha_ItemButter1 0.90 0.23 0.45 1.34 1.00 928
## logalpha_ItemArm1 1.12 0.22 0.69 1.55 1.00 925
## logalpha_ItemShore1 0.85 0.22 0.42 1.28 1.00 999
## logalpha_ItemLetter1 0.18 0.25 -0.31 0.65 1.00 1303
## logalpha_ItemQueen1 0.49 0.23 0.04 0.94 1.00 1056
## logalpha_ItemCabin1 0.06 0.25 -0.45 0.55 1.00 1369
## logalpha_ItemPole1 -0.88 0.40 -1.77 -0.20 1.00 2551
## logalpha_ItemTicket1 0.03 0.25 -0.49 0.53 1.00 1304
## logalpha_ItemGrass1 -1.96 0.56 -3.18 -1.01 1.00 6534
## logalpha_ItemEngine1 -2.56 0.53 -3.67 -1.61 1.00 5711
## logalpha_ItemTicket2 0.06 0.32 -0.62 0.64 1.00 1832
## logalpha_ItemCabin2 0.56 0.24 0.09 1.04 1.00 1170
## logalpha_ItemButter2 0.55 0.24 0.06 1.01 1.00 1276
## logalpha_ItemShore2 -0.68 0.40 -1.55 -0.00 1.00 2651
## logalpha_ItemEngine2 0.68 0.23 0.21 1.13 1.00 1049
## logalpha_ItemArm2 0.50 0.24 0.03 0.96 1.00 1098
## logalpha_ItemQueen2 -1.44 0.60 -2.74 -0.44 1.00 5757
## logalpha_ItemLetter2 0.68 0.24 0.21 1.13 1.00 1069
## logalpha_ItemPole2 -0.90 0.57 -2.19 0.05 1.00 6077
## logalpha_ItemGrass2 0.06 0.31 -0.60 0.62 1.00 1849
## logalpha_ItemQueen3 0.01 0.26 -0.52 0.51 1.00 1487
## logalpha_ItemGrass3 0.72 0.23 0.28 1.17 1.00 1094
## logalpha_ItemArm3 0.47 0.23 0.00 0.92 1.00 1235
## logalpha_ItemCabin3 -0.46 0.34 -1.19 0.16 1.00 2653
## logalpha_ItemPole3 0.67 0.23 0.22 1.12 1.00 1083
## logalpha_ItemShore3 -0.01 0.25 -0.50 0.47 1.00 1288
## logalpha_ItemButter3 -0.46 0.32 -1.14 0.15 1.00 2162
## logalpha_ItemEngine3 0.32 0.25 -0.19 0.79 1.00 1296
## logalpha_ItemTicket3 -0.28 0.27 -0.82 0.23 1.00 1621
## logalpha_ItemLetter3 -0.31 0.26 -0.85 0.19 1.00 1611
## Tail_ESS
## beta_LocDep.1 6268
## beta_LocDep.2 3966
## beta_ItemButter1 6424
## beta_ItemArm1 5368
## beta_ItemShore1 5635
## beta_ItemLetter1 6349
## beta_ItemQueen1 5282
## beta_ItemCabin1 5895
## beta_ItemPole1 5968
## beta_ItemTicket1 5828
## beta_ItemGrass1 5962
## beta_ItemEngine1 5980
## beta_ItemTicket2 6252
## beta_ItemCabin2 5981
## beta_ItemButter2 6355
## beta_ItemShore2 5227
## beta_ItemEngine2 5841
## beta_ItemArm2 5167
## beta_ItemQueen2 5713
## beta_ItemLetter2 5898
## beta_ItemPole2 5614
## beta_ItemGrass2 6343
## beta_ItemQueen3 5262
## beta_ItemGrass3 4964
## beta_ItemArm3 5280
## beta_ItemCabin3 5066
## beta_ItemPole3 6364
## beta_ItemShore3 5432
## beta_ItemButter3 5792
## beta_ItemEngine3 6407
## beta_ItemTicket3 5931
## beta_ItemLetter3 4896
## logalpha_LocDep.1 5430
## logalpha_LocDep.2 4819
## logalpha_ItemButter1 1579
## logalpha_ItemArm1 1319
## logalpha_ItemShore1 1852
## logalpha_ItemLetter1 3075
## logalpha_ItemQueen1 2364
## logalpha_ItemCabin1 2752
## logalpha_ItemPole1 3917
## logalpha_ItemTicket1 2704
## logalpha_ItemGrass1 5089
## logalpha_ItemEngine1 5260
## logalpha_ItemTicket2 3250
## logalpha_ItemCabin2 2403
## logalpha_ItemButter2 2714
## logalpha_ItemShore2 4175
## logalpha_ItemEngine2 2169
## logalpha_ItemArm2 2457
## logalpha_ItemQueen2 5519
## logalpha_ItemLetter2 2346
## logalpha_ItemPole2 5342
## logalpha_ItemGrass2 3961
## logalpha_ItemQueen3 3382
## logalpha_ItemGrass3 2150
## logalpha_ItemArm3 2540
## logalpha_ItemCabin3 4431
## logalpha_ItemPole3 2528
## logalpha_ItemShore3 2422
## logalpha_ItemButter3 4092
## logalpha_ItemEngine3 2654
## logalpha_ItemTicket3 3381
## logalpha_ItemLetter3 3432
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Item Covariates
mcmc_plot(TwoPL_itmex, type = "nuts_divergence")
mcmc_plot(TwoPL_itmex, type = "nuts_treedepth")
mcmc_plot(TwoPL_itmex, type = "trace", variable = "b_beta", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_itmex, type = "trace", variable = "b_logalpha", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_itmex, type = "trace", variable = "sd_", regex = TRUE)## No divergences to plot.
mcmc_plot(TwoPL_itmex, type = "rhat_hist", binwidth = 0.0001)
mcmc_plot(TwoPL_itmex, type = "neff_hist", binwidth = 0.1)
The 2PL model with all item covariates demonstrated no evidence of estimation concerns that would raise concerns for the validity of the results. Since there are no validity concerns, we can look at the results from the model overall.
mcmc_plot(TwoPL_itmex, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta", regex = TRUE)
mcmc_plot(TwoPL_itmex, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha", regex = TRUE)
pp_check(TwoPL_itmex, ndraws = 50, type = "bars")
pp_check(TwoPL_itmex, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))
Like the other 2PL models, this model estimates responses with high accuracy. The following general model summary integrates basic model validity statistics and posterior summaries for additional parameters. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.
summary(TwoPL_itmex)## Family: bernoulli
## Links: mu = logit
## Formula: Resp ~ beta + exp(logalpha) * theta
## theta ~ 0 + (1 | ID)
## beta ~ 1 + Time.c + poly(ItemPos, 2) + FreqSTX + Concrete + Diversity + AoA + BOI + Phonemes
## logalpha ~ 1 + Time.c + poly(ItemPos, 2) + FreqSTX + Concrete + Diversity + AoA + BOI + Phonemes
## Data: df_long (Number of observations: 36570)
## Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
## total post-warmup draws = 8000
##
## Group-Level Effects:
## ~ID (Number of levels: 1219)
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept) 0.75 0.47 0.13 1.91 1.00 12602 4763
##
## Population-Level Effects:
## Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_Intercept -0.04 0.02 -0.09 0.00 1.00 6323
## beta_Time.c1 0.97 0.03 0.91 1.03 1.00 12657
## beta_Time.c2 1.34 0.03 1.28 1.40 1.00 13518
## beta_polyItemPos21 12.46 1.59 9.27 15.58 1.00 21398
## beta_polyItemPos22 29.94 1.53 26.95 33.00 1.00 18744
## beta_FreqSTX -0.35 0.02 -0.38 -0.31 1.00 11432
## beta_Concrete -0.40 0.02 -0.45 -0.35 1.00 8659
## beta_Diversity -0.21 0.02 -0.26 -0.17 1.00 8776
## beta_AoA -0.17 0.02 -0.21 -0.13 1.00 14061
## beta_BOI 0.37 0.03 0.32 0.42 1.00 7587
## beta_Phonemes 0.11 0.02 0.08 0.15 1.00 10479
## logalpha_Intercept -0.22 0.70 -1.38 1.32 1.00 12706
## logalpha_Time.c1 0.22 0.06 0.10 0.35 1.00 9325
## logalpha_Time.c2 0.21 0.07 0.08 0.34 1.00 9057
## logalpha_polyItemPos21 -2.00 0.98 -3.92 -0.09 1.00 20742
## logalpha_polyItemPos22 -0.93 1.00 -2.89 1.02 1.00 18711
## logalpha_FreqSTX 0.16 0.04 0.08 0.24 1.00 8919
## logalpha_Concrete 0.06 0.06 -0.05 0.17 1.00 7797
## logalpha_Diversity -0.14 0.06 -0.26 -0.03 1.00 7449
## logalpha_AoA 0.10 0.05 -0.01 0.20 1.00 10494
## logalpha_BOI 0.19 0.05 0.09 0.29 1.00 7155
## logalpha_Phonemes -0.25 0.05 -0.34 -0.16 1.00 7099
## Tail_ESS
## beta_Intercept 6393
## beta_Time.c1 6739
## beta_Time.c2 6991
## beta_polyItemPos21 5766
## beta_polyItemPos22 5982
## beta_FreqSTX 6509
## beta_Concrete 6580
## beta_Diversity 6804
## beta_AoA 6318
## beta_BOI 6778
## beta_Phonemes 6825
## logalpha_Intercept 5044
## logalpha_Time.c1 7306
## logalpha_Time.c2 6336
## logalpha_polyItemPos21 5633
## logalpha_polyItemPos22 5620
## logalpha_FreqSTX 6709
## logalpha_Concrete 6695
## logalpha_Diversity 6553
## logalpha_AoA 6053
## logalpha_BOI 6850
## logalpha_Phonemes 6658
##
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).Final Model Details
This final section provides further details on the final item covariate model identified in this study. To begin with, some additional details regarding the model’s performance and fit are provided for readers. The manuscript provides summaries of coefficients as well as item and person fit statistics. In a previous section, this supplemental manuscript provided visual summaries of the coefficients as well. To further enrich the visual understanding of the final data, the following plots describe item and person fit further:
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
These two plots show the empirical distribution of the differences in log-likelihood of the predicted and observed data under the models. The first plot is grouped by items and reflects the item fit. In an ideal model, the distribution is normally distributed with a mean of 0. To address the impracticality of plotting all of these plots for each of 1219 participants, just the most extreme case (individual with worst fit) is shown. While the aim of this study was to examine predictors of item traits, it is still possible to visualize person traits (i.e., memory). The following plot shows the ordered distribution of latent trait estimates and their 95% credible intervals for each participant in the study:
The plots demonstrate that, on the whole, the estimates of a person’s latent trait is fairly wide. This is not entirely surprising given that the only estimate being given to the latent trait is a random intercept for each person. The explanatory item response theory model will be expanded in a future study to include factors that predict latent trait, which may reduce uncertainty in the latent trait measurement. The largest limiter to higher accuracy in the latent trait estimate is the fact that memory is being measured with just 10 items repeated only 3 times, so person predictors are likely not going to be a significant source of error reduction. It is also potentially useful to visualize all the conditional effects from the final model. These plots are shown next:
There are also some general test descriptives that are usually helpful to examine. These include test reliability, the expected score function, and test information. The following plots are these visualizations in their respective order:
## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'
## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.## `summarise()` has grouped output by 'Theta', 'Trial'. You can override using the `.groups` argument.
## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.
## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.
## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.
Within IRT, information and reliability are very closely related. It is clear that these plots both have generally the same shape with peaks in their functions around slightly below average scores. As is clear from both of these scales, the CERAD word list is not ideal for measuring individuals with either very low or very high memory abilities. In dementia/mild cognitive impairment research, this particular trait may be favorable as the goal is often to characterize with greater precision mildly impaired memory with relatively little need for carefully measuring extremes. The expected score function is a useful visual analogue for the relationship between raw score and latent trait estimate. The relationship is clearly monotonic and overall very gradual, which means that the raw scores do span a relatively wide range of latent trait values.